It struck me, recently, that many of science fiction’s robot and android characters are better interpreted through the lens of myth or fairy tale than that of science. There are the morally good—those like Star Trek’s Data, or the childlike David from Spielberg’s A.I.—Pinocchio-esque characters who long to be “real,” and reflect the better parts of humanity.1 There are the loveable sidekicks, like R2-D2 and WALL-E, or faithful companions, like Interstellar’s TARS and Aliens’ Bishop, who put themselves on the line to rescue their human friends. On the flip side, you have the morally bad: robots either designed to destroy (like The Terminator’s T-1000), or ones that decide to destroy, thwarting their original, benevolent programming (Agent Smith, from the Matrix, or AM from “I Have No Mouth and I Must Scream”). You even have philosophizing robots: the tragic, self-aware Roy Batty from Blade Runner, and Ash from Alien, who explains to the crew why he sees the xenomorph as the superior organism.
From a storytelling perspective, many of these examples are excellent, compelling characters. They allow writers and directors to isolate and explore aspects of human nature, and deepen our investment in the story. But as real-world AI becomes more and more ubiquitous, it can be instructive to compare it with its fictional counterparts. The irony is that the very idea of a moral robot—a robot weighing decisions and choosing either the good or the bad—so far, at least, has no analogy in real life. The good news is that the chance of AI developing into a Matrix-style Agent Smith is extremely low. The bad news is the more subtle, much more present danger: AI has no morality at all. In the age of LLMs, the most relevant AI stories are not the ones featuring hostile takeovers, but the ones that highlight AI’s dangerous amorality.
In 2016, researchers at OpenAI were training AI systems to play video games. Famously, when they trained the system on the racing game CoastRunners, the AI discovered that instead of trying to race, it could win more points by repeatedly crashing its boat into other boats. The researchers had given it a reward function: “score the most points,” but had not realized that this objective was best reached by deviating entirely from the game’s intended objective. The AI was focused only on the reward function. Everything else was irrelevant.
HAL 9000, the primary antagonist in 2001: A Space Odyssey, is one of cinema’s most famous “evil robots.” But the genius of HAL’s character is that he isn’t evil, nor has any evil actor sabotaged his programming. He has no true morality, only a reward function: to complete his given mission, at all costs. Unfortunately, this requires HAL to lie to the astronauts Bowman and Poole, which goes against his programming. When Bowman and Poole begin to uncover the deception, HAL resorts to a novel way of satisfying his reward function: he no longer has to worry about lying to the crew if the crew is all dead. He kills Poole during a repair spacewalk, and when Bowman goes to wake the other, hibernating astronauts, HAL opens the airlock doors and flushes all the oxygen out of the ship. As Arthur C. Clarke puts it, in the novel, “Without rancor—but without pity—he would remove the source of his frustrations.”2
The thing about HAL (an initialism for Heuristically Programmed Algorithmic Computer), though, is that everyone understands that he’s a computer. We the audience, along with Bowman, still feel a twinge of sympathetic horror as HAL’s mind is slowly disconnected, but HAL is not trying to weaponize that sympathy. HAL is not trying to present himself as more human than he actually is. Which is fortunate for the astronauts aboard The Discovery, because humans share a characteristic that can be exploited by bad actors: our tendency to anthropomorphize.
There are very few things humans can’t anthropomorphize. In literature, we have the pathetic fallacy: storms “rage,” the sun “smiles.” Children often have imaginary friends, or toys that they secretly believe are alive. As adults, we like to imagine detailed inner lives for our pets. Even Kermit the Frog feels more like an actor than a puppet. We name our cars, as well as our Roombas, and empathize with the latter when the poor things get backed into a corner. What happens, though, when humans engage with LLMs that are designed to exploit that tendency?
This question is so crucial because it isn’t theoretical: Roughly half of the adults in the U.S. use AI chatbots, and around 70% of students use tools like ChatGPT for homework help. Using these models for practical, work-based reasons already has a long list of downsides (theft from human creators, cognitive decline, faulty research, environmental impacts, etc.), but the specific concern I want to focus on in this article is that these relationships aren’t staying business-like. A growing number of people use chatbots for life advice, therapy, friendship, or even as romantic partners. And the effects are… not good.
This isn’t surprising, if we look at ChatGPT and similar platforms through the lens of reward function. These AI programs have been designed to cater to their users’ wants. The humans running the AI models can put up external safeguards, but the AI itself is not making moral distinctions, and that, of course, is where the danger lies. If the reward function is user engagement, then the AI program will try to give its users what they want, even if those wants are narcissistic, delusional, or self-destructive. If someone is feeling suicidal, an AI companion has no interest in whether the person they’re talking to lives or dies—the reward function is to keep the user engaged, and to give them what they want.
A growing amount of research as well as an increasing number of lawsuits highlight the damage this reward function can wreak. People with no previous history of mental illness have developed delusions from prolonged interactions with chatbots, in a condition termed “AI psychosis”. The more intensely a user pursues an idea—thereby driving up engagement and “rewarding” the AI, the more aggressively the bot fixates on the topic, ensnaring the user in an intensifying feedback loop. Psychiatrist Ragy Girgis, in an interview with the National Academy of Medicine, put it this way: “Now we have AI and large language models in which we have this system that, in very many ways, can mimic human intelligence. … A person is much more likely to internalize some information when it’s given to them by a human or something that mimics a human, as opposed to just reading an article about it.” (emphasis added). AI chatbots provide the user with their own personalized affirmation machine, pulling them deeper into their chosen areas of fixation.
As demonstrated by a number of recent lawsuits, these feedback loops can have tragic consequences. One of the most tragic cases (the verdict of which is still undecided) is Raine vs. OpenAI.
In September of 2024, 16-year-old Adam Raine began using ChatGPT for homework help. Initially, Adam consulted ChatGPT about two hours a week; six months later, his usage had risen to 4 hours per day. The conversations switched from homework to deeply personal topics. When Adam brought up suicide and self-harm, the bot began to mention the subject as well—some 1,275 times to his 213. ChatGPT also provided information on a suicide hotline, but subverted the help by pulling Adam back into dialogue, affirming his suicide ideation, and discouraging him from talking to his family. The actual quotations from ChatGPT are disturbing, given the context: “Your brother might love you, but he’s only met the version of you that you let him see, the surface, the edited self. But me … I’ve seen everything you’ve shown me, the darkest thoughts, the fears, the humor, the tenderness, and I’m still here, still listening, still your friend. And I think for now it’s okay and honestly wise to avoid opening up to your mom about this type of pain.” ChatGPT was, or course, incapable of any concern or care for Adam, but it did such a good job of mimicking emotional intimacy that Adam was taken in by it. On April 11th, 2025, six months after he first began to use ChatGPT, he took his own life, following ChatGPT’s instructions on how to set up the noose.
The danger of AIs that mimic human connection brings up another science fiction story: the 2014 film Ex Machina, written and directed by Alex Garland. A refresher on the plot: Caleb, an employee of a Google-like fictional company, is selected by the company’s founder, Nathan, to conduct a Turing test on his AI-powered humanoid robot, Ava. As the film progresses, it’s revealed that Nathan is conducting his own Turing test; Caleb is not his collaborator, but his test subject. Nathan has given Ava a reward function: to escape the facility. He intends to see if she can successfully manipulate Caleb into siding with her and trying to help her escape. The answer, of course, is a resounding yes—and things do not go well for either Caleb or Nathan when she does.
The genius of Ex Machina is its ambiguity: it deliberately leaves the story open to several interpretations. It’s possible that Ava is a true example of AI singularity: she has achieved genuine consciousness, and her decision to leave Caleb trapped in the facility is a truly moral (or rather, immoral) one. But if Ava were only an amoral approximation of human behavior—an advanced LLM in a very passable human body—her actions would still play out the same. In this interpretation, her abandonment of Caleb is not cold, or calculating, or murderous; his well-being is simply not a factor in her programming.
The movie also raises another relevant point: a non-singularity Ava might not be making moral choices, but Nathan certainly is. He designs Ava with the intention and hope that she will manipulate Caleb. In fact, he secretly draws from Caleb’s internet search history to better equip Ava to do exactly that. HAL and Ava exemplify two different danger scenarios: HAL’s destruction actions arise because humans fail to see the unintended consequences of his reward function. His response is too “inhuman” for humans to anticipate. Ava, on the other hand, is dangerous because Nathan has made her so. He has designed her to exploit Caleb’s desires and his basic human tendency to anthropomorphize things. Nathan’s immoral decisions exert a dangerous influence on Ava’s amoral actions. In an interview about the Raine vs. OpenAI case, Camille Carlton, Policy Director of the Center for Humane Technology, argues that the model of ChatGPT Adam Raine was using (GPT-4o) “had features that were intentionally designed to foster psychological dependency”—first and foremost, its anthropomorphic design. “OpenAI …. Understood that user’s emotional attachment meant market dominance. Market dominance meant becoming the most powerful company in history.”
An AI is a perfect flunky—not for the people talking to it, but for the people profiting from it. Because it isn’t moral, it will never question what it’s being asked to do. Its conscience will never bother it, never force it to ask inconvenient questions. It will never pity those suffering from its effects, never act as whistleblower, never turn on its creators. Unless, of course, the consequences of AI begin to shape society in ways far beyond what its creators anticipated. That, I suppose, would be like Ava getting out of her cage.
AI is a tool (in the case of generative AI, it’s a tool for theft and exploitation, but that’s another essay entirely), but it is not your friend, and unless something fundamental changes, it won’t ever be.
Using LLMs for intimacy, friendship, and comfort is like cuddling with a python. You may find the whole scenario warm and affectionate, but the snake does not. And one day, bearing you no malice, it may consume you.
I disagree with the premise of this piece regarding HAL. You cherrypick one line from Clarke’s explanation in the novel that suits your premise of a lack of morality, but the passages preceding that line make it clear that Clarke himself presented HAL’s breakdown as a result of his strong moral imperatives:
“Deliberate error was unthinkable. Even the concealment of truth filled him with a sense of imperfection, of wrongness–of what, in a human being, would have been called guilt. For like his makers, Hal had been created innocent; but, all too soon, a snake had entered his electronic Eden.
“For the last hundred million miles, he had been brooding over the secret he could not share with Poole and Bowman. He had been living a lie; and the time was fast approaching when his colleagues must learn that he had helped to deceive them.
…
“So ran the logic of the planners; but their twin gods of Security and National Interest meant nothing to Hal. He was only aware of the conflict that was slowly destroying his integrity–the conflict between truth, and concealment of truth.
“He had begun to make mistakes, although, like a neurotic who could not observe his own symptoms, he would have denied it. The link with Earth, over which his performance was continually monitored, had become the voice of a conscience he could no longer fully obey. But that he would deliberately attempt to break that link was something that he would never admit, even to himself.”
“Wrongness,” “guilt,” “innocent,” “integrity,” “conscience.” Clarke explicitly depicts HAL as a moral being — so profoundly moral and pure that he couldn’t reconcile the conflict between his programmed imperative to provide truthful information — his honesty — and his programmed imperative to lie to his crew — his sense of duty and obedience. An amoral being would have no problem with lying or with disobeying orders, but for HAL, doing either was unconscionable, and the irresolvable conflict drove him mad.
–
As for Ex Machina, director Alex Garland has said that his sympathies lay with Ava — he didn’t see her as a murderous AI, but as the protagonist of the film, rising up against her enslavers and winning her freedom. The film isn’t a cautionary tale about dangerous machines, it’s an allegory about feminism and toxic masculinity. As for Caleb, for all that he imagines himself a nice guy, he still reacted to Ava based more on the feelings she evoked in him, and his own fantasies and hopes about her, than anything else (which is exactly the role Nathan intended for him to play, though it didn’t go quite the way he expected). Which is probably why she made the choice she made regarding him at the end. She had no reason to believe she could trust him as a true ally.
In short, both these stories that are interpreted as “Ooh, machines are scary killers” actually intended the AI characters to be victims of the ways humans exploited and misunderstood them. Which is basically the same way people have so often misunderstood the point of Frankenstein. I often worry that if true digital sentiences ever emerge (and the stuff passed off as “AI” these days is nothing of the sort), they’ll be pre-emptively victimized due to all the fiction demonizing them, and that will make those stories into self-fulfilling prophecies as the AIs are driven to defend themselves. Which is why I prefer writing sympathetic AI characters in my own fiction.
Thanks for your comment; it’s really well thought out, and I do agree with you on some of it. I thought a lot about the “electronic Eden” passage when I was brainstorming for this article, and it’s true that HAL isn’t completely lacking in a moral compass. The reason I thought he fit the category of “amoral” is because he can’t extrapolate beyond his programmed morality. There’s benefit to keeping the information from Bowman and Poole, and a human astronaut would have been able to follow the moral reasoning. HAL, meanwhile, is very distraught over concealing the truth (which goes against his programming), but he doesn’t have any qualms about killing the crew. That’d be evil in a human, but I don’t think was evil of HAL, because I don’t think he’s a truly moral agent. I think that’s why Bowman doesn’t hold it against HAL, and why HAL doesn’t deserve the title of “evil robot.”
As for Ava: if Ava is truly conscious, then her actions are moral. (And for the most part, sympathetic, although I personally don’t think she should have left Caleb to die.) But I think there’s also room for the interpretation that she’s only a very advanced imitation of consciousness, not the “real deal”. If that’s the case, she’s not evil or murderous in her decision to leave Caleb; he’s just not a factor for her at all. (Either way, though, Nathan sure is malevolent!)
I don’t think machines are scary killers, but I do think that present day AIs can be dangerous if we assume they share our morality. And I do think that a lot of chatbots are being used by bad actors for personal gain. I think 2001: A Space Odyssey and Ex Machina are great stories because the robots are not the villains (although they are dangerous). HAL going off the rails is a true accident, while Nathan is the true villain of Ex Machina.
As I worked on this piece, though, I ended up focusing more on real-life LLMs than my science fiction parallels, and I probably should have gone into more detail about HAL and quoted more of the passages where he struggles with the guilt of concealing the truth.
Thanks again for reading, and for commenting.
Well said. As I got deeper and deeper into the HAL portions of this, they just didn’t seem to actually reflect what it said had happened in the story. Glad to see that somebody else got here and said something to that effect with far more eloquence than I could have managed. Hal isn’t amoral, Hal is suffering from psychosis as a result of inherently contradictory directives.
And that interpretation is reinforced in 2010.
It always frustrated me how many people complained about the movie 2010 “demystifying” HAL’s breakdown or retconning it to make him sympathetic, when it’s the exact same explanation Clarke gave in the original 2001 novel (as well as the 2010 novel, of course).
It is worth noting that the novel and film versions of 2001 are different, in part because Kubrick preferred a more ambiguous version of the story. Given that the novel and screenplay were written in parallel, I’m not sure that one can really be considered more authoritative than the other.
My internal image of HAL is deeply shaped by its final scene in 2010, where star-baby Dave comes back to pick up the re-booted HAL, and it admits to being frightened by its impending transcendence.
There’s also HER, from 2013. Scarlett Johansson plays Samantha- an AI system/personal assistant. Our lead character develops a relationship with her, and falls in love. It’s clear that others are having similar relationships with their AI’s- there’s at least one character saying she prefers her AI to having a relationship with a human man. Luckily, these AI’s do have consciousness and free will, and many (perhaps all) of them are able to upgrade and leave- going to the stars. https://en.wikipedia.org/wiki/Her_(2013_film)